Non-GPU-resident symmetric indefinite factorization

Department of Electrical Engineering and Computer Science, University of Tennessee, Knoxville, USA Correspondence Ichitaro Yamazaki, Department of Electrical Engineering and Computer Science, University of Tennessee, Knoxville, USA. Email: [email protected] Funding information National Science Foundation NVIDIAMatrix Algebra for GPU andMulticore Architectures (MAGMA) for Large Petascale Systems, Grant/Award Number: ACI-1339822 Summary We study various algorithms to factorize a symmetric indefinite matrix that does not fit in the core memory of a computer. There are two sources of the data movement into the memory: one needed for selecting and applying pivots and the other needed to update each column of the matrix for the factorization. It is a challenge toobtainhighperformanceof suchanalgorithmwhen the pivoting is required to ensure the numerical stability of the factorization. For example, when factorizing each columnof thematrix, a diagonal entry,which ensures the stability,mayneed tobe selectedasapivot among the remainingdiagonals, andmoved to the leadingdiagonalby swapping both the corresponding rows and columns of the matrix. If the pivot is not in the core memory, then it must be loaded into the core memory. For updating the matrix, the data locality may be improved by partitioning the matrix. For example, a right-looking partitioned algorithm first factorizes the leading columns, calledpanel, and thenuses the factorizedpanel toupdate the trailing submatrix. This algorithm only accesses the trailing submatrix after each panel factorization (instead of after each column factorization) and performs most of its floating-point operations (flops) using BLAS-3, which can take advantage of the memory hierarchy. However, because the pivots cannot be predetermined, the whole trailing submatrix must be updated before the next panel factorization can start. When the whole submatrix does not fit in the core memory all at once, loading the block columns into the memory can become the performance bottleneck. Similarly, the left-looking variant of the algorithm would require to update each panel with all of the previously factorized columns. This makes it a much greater challenge to implement an efficientout-of-core symmetric indefinite factorizationcomparedwithanout-of-corenonsymmetric LU factorization with partial pivoting, which only requires to swap the rows of the matrix and accesses the trailing submatrix after each in-core factorization (instead of after each panel factorization by the symmetric factorization). To reduce the amount of the data transfer, in this paper we uses the recently proposed left-looking communication-avoiding variant of the symmetric factorization algorithm to factorize the columns in the core memory, and then perform the partitioned right-looking out-of-core trailing submatrix updates. This combinationmay still require to load the pivots into the corememory, but it only updates the trailing submatrix after each in-core factorization, while the previous algorithm updates it after each panel factorization.Although these in-core and out-of-core algorithms can be applied at any level of the memory hierarchy, we apply our designs to theGPUandCPUmemory, respectively.We call this specific implementation of the algorithm a non–GPU-resident implementation. Our performance results on the current hybrid CPU/GPU architecture demonstrate that when the matrix is much larger than the GPU memory, the proposed algorithm can obtain significant speedups over the communication-hiding implementations of the previous algorithms.